In recent years many of the functions of harvesting machines normally controlled manually by an operator have been automated. Audible and visual sensing by the operator have been replaced with optical, sonic, magnetic, radio frequency and other types of sensors. Microprocessors, operating in response to conditions sensed by the sensors, have replaced manual operator control of the mechanical functions. However, it is still necessary to have an operator for steering the harvester to move along a crop line.
Human capability is a key limitation in the efficiency of harvesting. The harvester is operated most efficiently at maximum speed and with one end of the cutter mechanism slightly overlapping the crop line between cut and uncut crop. If there is is no overlap, an uncut strip of crop is left in the field. On the other hand, if the overlap is too great the maximum width of crop is not cut.
Currently available harvesters are operable at ground speeds of up to 4 to 5 miles an hour and future harvesters are being designed for operation at higher speeds. Although an operator may easily steer a harvester along a crop line at speeds of 4 to 5 miles an hour, the constant attention required to accomplish this is extremely tiring and an operator can not maintain maximum harvesting efficiency at this speed for long periods of time. Thus efficiency of utilization of the harvester could be increased by providing a form of "cruise control" which automatically steers the harvester along a crop line at the maximum harvester speed, with operator intervention being required, at the most, only as the harvester approaches the end of a crop field.
Apparatus that can be utilized for detecting the crop line can be divided into two categories: range-based methods which attempt to detect the height difference between the cut and uncut crop; and vision-based methods which attempt to detect appearance differences between the crop on the cut and uncut side of the crop line. Range-based methods can include a scanning laser rangefinder, a laser light stripper, a stereo system, and an optical flow method. Vision based methods can include a feature tracking technique and segmentation on the basis of texture, intensity, and color.
During development of the present invention consideration was given to a configuration utilizing a two-axis scanning laser rangefinder. This sensor scans a laser beam over a scene and measures the time it takes for the reflected light to return to the sensor. The system returns depth information over a 360 degree azimuth and a 40 degree elevation field of view. The sensor makes about ten sweeps a second and thus takes roughly twenty seconds to generate a full 2048 by 200 range image. This scanner is capable of detecting height differences of only a few cm at a range of 20 meters, and in field tests, the height difference between the cut and uncut crop showed up quite clearly. The drawbacks for this system include slow cycle time, high cost, and concerns about mechanical reliability.
Another alternative system considered for evaluation was a laser light striping sensor. Light striping systems have two components: a light source, and a camera tuned to detect only the frequencies emitted by the light source. Typically, a laser beam is sent through a cylindrical lens which spreads the linear beam into a plane. This plane intersects the scene in an illuminated line. Distances to all the points on the line can then be estimated by triangulation.
The light striper has no moving parts, requires only minimal computational power, and is potentially inexpensive. However, since harvesters function outdoors, a prohibitively powerful laser would be needed to gather range information at the necessary distance (around 5 meters) in order to avoid being washed out by sunlight. The light striper system also has the disadvantage that it returns only a linear array of depths, so that only one point along the cut line can be detected at a time.
Stereo cameras provide another possible alternative range-based approach to detecting the crop cut line, based on another triangulation-based method. Depth information is obtained by viewing the same object from two or more different viewpoints simultaneously, as with the human vision system. In this application, extremely precise depth information is not needed. We therefore investigated two camera systems with relatively small baselines (between 6 inches and two feet). The storm stereo algorithm, as defined in an article for the 1993 IEEE Conference on Computer Vision and Pattern Recognition, by Bill Ross, entitled "A Practical Stereo Vision System", was used to compute the correspondences. Although it was found that an alfalfa field contains sufficient texture to solve the correspondence problem, computational speed remains a major concern with stereo-based tracking.
An alternative vision-based technique is defined as a window-based feature tracking method, as described in a 1991 Carnegie-Mellon Technical Report published by C. Tomasi and T. Kanade, entitled "Detection and Tracking of Point Features". Starting with an image of the crop cut line, a set of small (approximately 20.times.30 pixel) windows which overlapped the cut line boundary were selected by hand. These windows were then input as features to the algorithm, and then tracked from one image to the next. In this manner, it was possible to track the crop line across a sequence of twenty images. However, this method still requires that the initial feature windows be chosen along the crop line.
Yet another alternative method which was considered utilized a local 2-D fourier transform operator as the basis for texture-based segmentation, as described in a 1994 article entitled "Segmenting textured 3D surfaces using the space/frequency representation" in Spatial Vision, Volume 8, No. 2. The intent of this method was to locate a spatial frequency band with a substantial difference between the cut and uncut crop. Unfortunately, however, preliminary testing failed to show any clear evidence of such an indicator.
A visual examination of some crop line images showed that portions of the image containing uncut crop were substantially darker and a slightly different hue than the portions containing uncut crop. Of the two, the color difference is generally more pronounced, and is largely due to the exclusive presence of a leaf canopy on the uncut side of the crop line. Due to the consistency of this color effect, the relative robustness of the sensor, and the lower sensor cost, a vision-based system using color segmentation techniques presented the most promising method to pursue for cut line tracking.
Vision-based guidance of agricultural vehicles is not a new idea, and others have investigated the perception problem which is involved. For instance, J. F. Reid and S. W. Searcy, in an article for the ASAE in Nov/Dec, 1988, entitled "An Algorithm for Separating Guidance Information from Row Crop Images", describe a method of segmenting several different crop canopies from soil by intensity threshholding. They do not, however, actually use the algorithm to guide a vehicle. M. Hayashi and Y. Fujii, in an article for the USA-Japan Symposium on Flexible Automation entitled "Automatic Lawn Mower Guidance Using a Vision System", have used smoothing, edge detection and a Hough transform to guide a lawn mower along a cut/uncut boundary. Their algorithm only finds straight boundaries, however, and they give no mention of the speed at which they are able to accomplish this task. Gerhard Jahns presents a review of automatic guidance techniques for agricultural vehicles in a 1983 ASAE paper entitled "Automatic Guidance in Agriculture: A Review". To applicants' knowledge, the work described in this paper is the only system which has ever been successfully used to guide a harvesting machine.